System and server for analyzing and integrating data collected by an electronic device

ABSTRACT

A system for analyzing data received from at least one mobile electronic device comprises a mobile electronic device and a server. The mobile electronic device includes a sensor configured to collect video, audio and/or imagery data, and the mobile electronic device is configured to initiate collection of data based on a command from a user of the mobile electronic device. The server includes a receiver configured to receive the video, audio and/or imagery data collected by the sensor of the mobile electronic device and a processing system. The processing system of the server includes a processor and is configured to, when a condition is met in accordance with a rule established on the server, issue a command to turn on the sensor of the mobile electronic device and enable the mobile electronic device to transmit the video, audio and/or imagery data collected by the sensor to the server, without any input by the user of the mobile electronic device. In addition, the video, audio and/or imagery data are collected by the sensor at a location of the mobile electronic device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of application Ser. No. 14/561,564,filed Dec. 5, 2014, which claims priority under 35 U.S.C. § 119(e) toU.S. Application No. 61/912,337, filed Dec. 5, 2013, U.S. ApplicationNo. 61/912,944, filed Dec. 6, 2013, and U.S. Application No. 61/914,755,filed Dec. 11, 2013, the entire content of each of which is incorporatedinto the present application by reference.

COPYRIGHT

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patentdisclosure, as it appears in the Patent and Trademark Office patentfiles or records, but otherwise reserves all copyright rightswhatsoever.

FIELD OF THE INVENTION

One aspect of the present disclosure relates to crowd sourcedcomputer-implemented methods and systems of collecting and transformingportable device data to cause actionable responses including machine tomachine (M2M) such as computer aided dispatch (CAD), analytic tools, orcommand and control (C2) tools; and/or machine to person (M2P) by one ormore human actors, such as an emergency first responder, a crimeinvestigation organization, public safety personnel, a private citizen,or a private security firm.

Another aspect of the present disclosure relates to crowd sourcedcomputer-implemented methods and systems of collecting and transformingportable device data computer-implemented methods and systems forcommunicating data to cause actionable responses including machine tomachine (M2M) such as computer aided dispatch (CAD), analytic tools, orcommand and control tools; and/or machine to person (M2P) to causeactionable responses by one or more human actors, such as public healthactors.

BACKGROUND OF THE INVENTION

Prior art systems in many cases require other specialized pieces ofhardware in addition to the portable device and/or require there to beperson in the loop that makes judgments about the data being received.Additionally many of those other systems merely receive alerts orupdates from some other sources of data or send a picture or text to a3rd party to create an alert. Other apps like the WAZE application(hereinafter “app”) are used in a crowd-sourced fashion to avoid publicsafety personnel (e.g., circumvent traffic cameras, radar checks, etc.)or to get basic situational awareness.

However, the WAZE app simply sends data collected by users to a publicwebsite for viewing by other users. The WAZE app does not include anyfeatures where a server analyzes the data and sends specificalerts/commands to individual users to create enhanced situationalawareness and/or to provide instructions.

Further, there exists a need to generate public health data by and fromthe public to the public in a time-sensitive fashion. Crowd-sourced Appsand services PatientsLikeMe, and 23 and Me focus on serving individualsin a peer-to-peer way but they do not serve the larger public good in amany-to-many or one-to-many manner. The World Health Organization (WHO)and the Center for Disease Control (CDC) all publish warnings andbulletins but all based on scientific field collected data notnear-real-time crowd sourced data that can be updated by userscontinuously.

SUMMARY OF THE INVENTION

The present invention broadly comprises crowd-sourcedcomputer-implemented methods and systems of collecting and transformingportable device data. One embodiment of the invention may be implementedas a system including an electronic device including a sensor configuredto collect data, the device configured to begin collection of data basedon a command from a user of the electronic device; and a serverconfigured to issue a command to the electronic device to turn on thesensor and transmit data collected by the sensor to the server withoutany input by the user of the electronic device when a condition is met.

Another aspect may be embodied as a system including an electronicdevice configured to collect data using a sensor; and a serverconfigured to receive data from the electronic device, to create a mapusing the data, and to transmit the map to the electronic device.

Still another aspect may be embodied as a system including an electronicdevice configured to collect data using a sensor; and a serverconfigured to receive the data from the electronic device, to create amap using the data, and to transmit warning information to theelectronic device.

BRIEF DESCRIPTION OF THE DRAWINGS

A full and enabling disclosure of the present subject matter, includingthe best mode thereof, directed to one of ordinary skill in the art, isset forth in the specification, which makes reference to the appendedfigures, in which:

FIG. 1 is a diagram of the structure of a system according to anexemplary embodiment of the present invention;

FIG. 2 is another exemplary system of the present invention;

FIGS. 3 and 4 illustrate exemplary screen shots for a client devicewhich may be part of the exemplary systems shown in FIGS. 1 and 2;

FIG. 5 illustrates a screen shot of an embodiment of the presentinvention where the client can display reported cases to a user on amap;

FIG. 6 illustrates a screen shot of an embodiment of the presentinvention where the client can display locations of chat participants;

FIG. 7 illustrates a screen shot of an embodiment of the presentinvention where the client can display a list of reported cases andprovide additional information on a selected case;

FIG. 8 illustrates a screen shot of an embodiment of the presentinvention where the client can display reported cases to a user on a mapwith additional information for a selected case;

FIG. 9 illustrates a screen shot of an embodiment of the presentinvention where the client can be manually set in a boss mode;

FIG. 10 illustrates a screen shot of an embodiment of the presentinvention where the client is currently in boss mode;

FIG. 11 illustrates another exemplary system according to the presentinvention;

FIG. 12 illustrates an exemplary server that may be part of the systemsshown in FIGS. 1, 2, and 11;

FIG. 13 shows a system according to another exemplary embodiment of theinvention; and

FIGS. 14-16 show exemplary screen shots of a client device that may beused with the embodiment shown in FIG. 13.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Reference is presently made in detail to exemplary embodiments of thepresent subject matter, one or more examples of which are illustrated inor represented by the drawings. Each example is provided by way ofexplanation of the present subject matter, not limitation of the presentsubject matter. In fact, it will be apparent to those skilled in the artthat various modifications and variations can be made in the presentsubject matter without departing from the scope or spirit of the presentsubject matter. For instance, features illustrated or described as partof one embodiment can be used with another embodiment to yield a stillfurther embodiment. Thus, it is intended that the present subject mattercovers such modifications and variations as come within the scope of thedisclosure and equivalents thereof.

FIG. 1 shows an exemplary embodiment of a system 10 in accordance withthe present invention. Server 20 receives data from a plurality of userdevices such as image and video data from device 30, voice and text datafrom device 40, temperature and other data from device 50, and location,altitude, and speed data from device 60. Thus device 30 includes asensor such as a microphone and a camera, device 40 includes a sensorsuch as a microphone and a keyboard for receiving text data that may beembodied by a touchscreen displaying the keyboard, device 50 includes asensor such as a thermometer, and device 60 includes a device such as aglobal positioning system (GPS) sensor. Devices 30, 40, 50, and 60 maybe smartphones, tablets, digital cameras, laptop or desktop computers orany other electronic device capable of collecting and transmitting thisdata. Further, any of the user devices 30, 40, 50, and 60 may includemore than one sensor, or all of the sensors listed above. In general,each of devices 30, 40, 50, and 60 will include at least one sensor, aprocessor, memory, a transmitter for transmitting the data to server 20,and a receiver for receiving data from server 20. Server 20 alsoincludes a processor, a memory, a transmitter for transmitting data todevices 30, 40, 50, and 60, and a receiver for receiving data fromdevices 30, 40, 50, and 60. Devices 30, 40, 50, and 60 will be referredto hereinafter as end point devices (EPDs).

In one embodiment, the EPDs are portable electronic devices that run oneof the Android®, iOS®, or Blackberry® operating systems. An app run bythe device performs the functions described herein as performed by theEPD. An exemplary portable electronic device is a wearable electronicdevice including a video camera and microphone described in U.S. patentapplication Ser. No. 13/212,686. This application is incorporated byreference herein. IN another embodiment, the EPD may be a wearable (bodyworn) health tracking device such as the FitBit®, Pebble®, Basis Peak®,etc.

Server 20 may include a complex event modeler 70 and a predictivemodeling tool 80 which analyzes the data received from the devices todetermine if the data received from the devices corresponds to an eventsuch as an emergency. The event can be a crime in progress, a severeweather event, or any emergency scenario where life or human/propertysecurity (e.g., child abduction, car break-in, arson, tornado, flashmob, etc.) is already or about to be imperiled. Server 20 manages andtransforms event files and automatically generates notifications,including machine to machine (M2M) notifications, using a computer aideddispatch (CAD) tool 90, analytic tools, or command and control tools;and/or machine to person (M2P) notifications to a private or publicactor 100 to respond to the event by sending an alert(s) to the actor100 with information about the event derived from the uploaded data. Theprivate or public actor can be an emergency first responder (lawenforcement, fire, and/or ambulance), a crime investigation organizationsuch as the FBI, public safety personnel, or a private security firm(such as hired for security at a sporting game like the Olympics, SuperBowl, or World Cup). The action taken by the actor can includedispatching one or more first responder(s), such as a fire truck, anambulance, or a police vehicle and associated first responder personnel,or causing an amber alert to be issued, for example.

Complex event modeler 70 is the analytic engine inside the server 20that allows thousands to millions of data feeds to come in from the EPDsand then alert on pre-defined thresholds. For example, if a fire is seenin a video the complex event modeler 70 will send an alert to thenearest fire department and send notices to EPD users in the immediatearea. In one embodiment, complex event modeler 70 may include theGeoVigilance tool commercially available from Transvoyant.

Predictive modeling tool 80 is the analytic engine inside the server 20that takes the alerts and data from the complex event modeler 70 andthen “predicts” the next likely group of scenarios. For example, a fireon or near a major thoroughfare would generate an alert that indicatestraffic will cause delays in the area and EPD users should planaccordingly. In one embodiment, predictive modeling tool 80 may includethe SilverEye tool commercially available from Knowmadics, the TotalInsight tool commercially available from Larus, or the Satellite ToolKit (STK) commercially available from Analytical Graphics, Inc.

In one embodiment, CAD tool 90 includes the SilverEye web-based softwareapplication commercially available from Knowmadics, Inc. running in aCAD center. SilverEye may be the device management software in thesystem 10 that all the EPDs running the app are connected to. SilverEyein a CAD center allows data from EPDs to be visualized securely andquickly without having to replace the current investment legacyhardware/software in the CAD center. For example, a computer in the CADcenter that has internet connectivity can visualize/playback imagery,video, and audio data from EPDs running the app as the data is collectedto supplement the traditional data collected from a 911 call—location,voice description, and identity (phone number).

Alerts may be generated by the server 20 or CAD tool 90 based on thedata received from the EPDs. For example, using SilverEye an operatorcan set an alert(s) based on certain conditions/groups of conditionsbeing met or exceeded—location, time, key words, weather, and/ortemperature etc. When the conditions set by the operator are met, theCAD tool 90 automatically generates an alert—machine-to-machine (M2M) orchanges a condition on another device. For example, a geographicboundary/geo-fence can be created and when say 10 or more objects enterthe boundary after 6 PM. Alerts may be generated by the server 20 orbased on the data received from the EPDs. What triggers an alert—usingSilverEye an operator can set an alert(s) based on certainconditions/groups of conditions being met or exceeded—location, time,key words, weather, temperature etc. that when the conditions are metautomatically generates an alert—machine-to-machine (M2M) or changes acondition on another device. For example, a geographicboundary/geo-fence can be created and when say 10 or more objects enterthe boundary after 6 PM, an alert can automatically be generated to asecurity guard to go check out the area for suspicious activity. In thiscase, an alert can automatically be generated by CAD tool 90 andtransmitted to a security guard 100 to go check out the area forsuspicious activity.

In another embodiment, server 20 or CAD tool 90 may generate alerts tobe sent to EPDs by linking multiple EPDs to other types of devices suchas cameras, audio recorders, trackers, seismic sensors, etc. Forexample, a geographic boundary can be set on the SilverEye controlsoftware so that when an EPD connected to system 10 enters, leaves,passes-by, etc. the geographic boundary an alert is generated which willenable a third party camera to track the EPD remotely without any humanin the loop. That camera data can then be sent automatically to anotherEPD connected to system 10.

In an exemplary embodiment shown in FIG. 2, EPD 150 is a smartphonecapable of collecting all of the above described data, which runs an appto perform these functions. The app allows selectable wirelesstransmission of a known and/or anonymous user's geographic locationcoordinates, audio, video, voice, text, temperature, velocity, oraltitude (or any other sensed data available on the EPD 150) to server20.

A user who witnesses an event can create a report on EPD 150 to uploadto the server 20. FIG. 3 shows exemplary screen shots of EPD 150creating a report. Screen 310 shows an opening menu for creating areport. Screen 320 allows a user to select a type of report (police,fire, weather, lost child, etc.). Screen 330 allows the user to remainanonymous, and add whatever type of data they have collected to thereport. The report can include a text summary of the incident the userwishes to report, and audio/video/photo attachments. The user identifiesthe type of alert, and the report, attachments are uploaded to theremote system, with the option to retain a copy of the report or totransmit without storing any data on the user's EPD. Multiple users (thecrowd) witnessing the same incident/event can upload reports and sensordata about the event to the same remote system in a crowd-sourcingmodel. Data previously collected on the EPD 150 by other existing appscan be added to a CASES/AGENT report. For example, an image that wastaken on an iPhone with the iOS® Camera app can be appended to a CASESreport which is sent to a CAD tool 90.

As shown in FIG. 4, screen 410 shows alerts that other users in thevicinity of the event, and who have authorized their EPDs to receivealerts, can receive from the remote system 20 about the ongoing event.Screen 420 allows the notified user to provide further data to server20. Screen 430 allows the user to notify the server 20 of the user's ownskillset or competency (e.g., law enforcement, firefighting, socialwork, medical training, search and rescue, crisis housing), and if theevent calls for a particular competency, the remote system canautomatically send alerts to all users with competencies relevant to theevent and who are located in the vicinity of the event information aboutthe event so that the user can utilize their competencies to amelioratenegative consequences caused by the event. Such users would haveauthorized the app to track their location so that the remote system cansend alerts only to those users located close to the event of interest,regardless of whether those users have witnessed the event or submitteda report.

The server 20 can also provide a list of previously submitted reports tothe EPD 150. As shown in FIG. 5, the EPD can display a map 540 withindicators 550 at each report location. Menu buttons 510-530 allow theuser to select whether the map will include indicators to show their ownlocation, other agents, and/or the report locations (“cases”). In theembodiment shown in FIG. 5, the user's own location is being shown, withthe map roughly centered on the user's location. Even if the user'slocation is turned off with button 510, the map may still be centered onthe user's location as a default state. The map shown in FIG. 5 is astreet map, but any local map is within the scope of the invention, suchas maps of stadiums as discussed below.

Further, the EPD can support a chat function which allows the EPD userto chat as shown in FIG. 6. The EPD can display the distance 630 anddirection 610 of a plurality of chat participants 620 so that the usercan directly gather further information about local events, or warnothers.

FIG. 7 shows that the EPD displaying a list of local cases, along withthe direction 710 and distance 720 to the location of the case. When auser selects a particular case, further information 730 is provided.This further information may include some or all of the data thereporting EPD provided to the server 20. This further information may bedisplayed on the map proximate the location of the case, as shown inFIG. 8. FIG. 8 illustrates an exemplary map 810 with case locationmarkers 820 and case information 830.

Any user can also authorize the EPD to turn any selected sensor on theEPD on or off (e.g., microphone, camera, GPS, accelerometer) and uploadthe selected sensor outputs in real time to the server 20. Further, byselecting the boss mode button 920 shown on screen 910 of FIG. 9, thiscan be done surreptitiously for the safety of the user. In this case, aninnocuous screen is displayed during data collection, such as theexemplary game display 1000 shown in FIG. 10. Any screen unrelated todata collection may be used to prevent a hostile person from seeing thatthe user is collecting and reporting data, possibly related to a crimebeing committed by the hostile person.

In another embodiment, server 20 issues a command to the EPD 150 toenter boss mode without any command by the user of EPD 150. In thisregard, a rule set can be established by server 20 based on conditionsbeing met that would automatically enable collection to occur on the EPD150 without the user having to do anything. In one embodiment, server 20can command each EPD 150 to start recording/streaming video whenever theEPD 150 was within 1 mile of a landmark such as the Washington Monument.In another embodiment, the server 20 may command every EPD 150 within aset distance of a reported case to begin recording sensor data and totransmit the sensor data to server 20.

The EPD also allows the user to select three levels of participation:anonymous in which the user uploads reports or sensor data anonymously,passive in which the user's personal identification information isreported with the sensor data uploaded, and remote control in which theuser allows the remote system to control one or more sensors on theuser's EPD for transmission to the remote system. The EPD can be placedin an invisible or surreptitious mode in which it will transmit sensordata in the background without conveying any human-discernible cues thatit is doing so. In this regard, FIG. 11 shows that a server 1110 cansend a command to device 1120 to collect and transmit data without theuser knowing. The data is sent to repository 1130 to be analyzed byserver 1110.

FIG. 12 shows an exemplary display of the CAD tool 90, which may beembodied by SilverEye™ software. The features of this tool are describedabove with respect to FIG. 1.

In another exemplary embodiment, the above described features may bedivided between two apps, the CASES app and the CASES AGENT app. CASESand CASES AGENT apps are distributed on EPDs with back end supportprovided through a cloud model controlled via an enterprise service bus(ESB). The primary CASES app turns EPDs into sensors and those sensorscan be used in a crowd-sourced fashion to help law enforcement, publicsafety, and defense personnel in a time of crisis or danger. The CASESecosystem involves the software app and the software back end datatransformation which occurs in the cloud as data from the EPDs isanalyzed and in the cloud. The CASES AGENT app has secondary featuresthat allow it to be used (turned on and off) remotely.

Primary features include:Philosophy of CASES and CASES AGENT=Collection−Transformation−Action

CASES and CASES AGENT are part of an ecosystem that includes adownloadable app which connects to a cloud based transformation enginewhich then sends machine to machine (M2M) and/or machine to person (M2P)alerts which cause action to occur in the real world.

It was designed primarily for everyday use as well as venue/eventspecific use. End users (civilians) can see an event and send data asquickly and easily as possible as an enhancement to public safety.

Data is received by the CASES back end processing capability in thecloud transforms the raw data feeds into a case.

The app can be customized by end users and white labeled for specificevents—such as the Superbowl, Olympics, World Cup, Grand Prix, etc. Inthose instances the actual seating chart of the venue could bedownloaded as an add on and users in the ecosystem can identify wherethey are sitting/standing so that when an event occurs the data theygenerate can be tied to a specific area within the event.

Uses the off-the-shelf/out-of-the-box capability of the EPDs to senddata—location, audio, video, text, temperature, speed, altitude and anyother data that can be collected by the EPD to the cloud for follow onanalysis, cataloguing, and distribution.

Quick way for average citizen to share observations from their EPDs.

Venue Specific downloads are available so that at an event CASES userscan let people know where they were sitting/standing etc. when an eventoccurred.

Everyday CASES users can register any particular skill set they havethat would make them more useful in an actual emergency so thatofficials would know what type of Good Samaritan support there was nearan emergency.

CASES reports can be shared with public safety and law enforcementpersonnel.

Directly to Law Enforcement, public safety, or to a “Cut out” serverwhich is accessible by personnel at a computer aided dispatch (CAD)center or public-safety answering point (PSAP), sometimes called“public-safety access point” (a call center responsible for answeringcalls to an emergency telephone number for police, firefighting, andambulance services). This CAD center may house the CAD tool 90 asdescribed above.

An enhanced version of CASES called CASES AGENT has all of the samecapability plus listed above plus:

The AGENT version can be remotely controlled by command and control (C2)software in server 20 to turn on/off the camera, audio and locationaldata streams from EPD 150 the AGENT version is hosted on, as shown inFIG. 11. This command to enter boss mode by the server 20 does notinvolve any input by the user of EPD 150, as discussed above.

AGENT Version has a panic button feature.

AGENT Version has a manually selected boss mode so that a user can makeit appear as if the app is not running if they had to turn their EPDover for forensic inspection, as shown in FIG. 10.

AGENT Version has a primary mission of information collection for publicsafety.

AGENT Version can be scheduled to turn and off based on time of dayand/or location.

The remote system can communicate with, for example, the FBI, the DEA,other law enforcement, public safety, or military operations.

Thus, the CASES and CASES AGENT app technologies combine crowd sourcingwith civic responsibility to create an ecosystem where moderntechnology—specifically the billions of dollars of investment in EPDsand the cloud—can be used to do good. It puts technology that is alreadyin the hands of ordinary citizens to work for the common good. Someadvantages of the CASES and CASES AGENT app include that it creates acentral application to process and fuse multiple types of data from EPDsand then easily send it from the EPD to the cloud with a simple buttons.

The CASES and CASES AGENT apps are designed to be customized so that itcan be licensed to a sponsor who becomes the sponsor of the app beingused at specific events such as the Olympics, etc. It can be customizedso that certain EPD features can be turned on and turned off incountries where data collection of this type is prohibited.

Additional advantages of the invention may include (this list is notexhaustive):

-   -   1. Single screen app interface—as opposed to 2-4 separate        applications with multiple interfaces, such as having a separate        app to track a phone, an app to take a picture, an app to record        audio, an app to record a video, or an app to chat.    -   2. Multiple date feeds from multiple EPD sensors—as opposed to a        user experience where each screen can only handle one feed at a        time.    -   3. Crowd sourced data inputs from social media—as opposed to        just getting one way notification alerts from a Rich Site        Summary (RSS) feed or broadcast.    -   4. Can be used as an information collection and transmission        tool in real time—as opposed to collecting data and then sending        it at a later date in response to an alert or after an event.        For example, the Boston Marathon Bombing had thousands of people        collecting images and video, but without any way to easily and        rapidly transmit that data to public safety and law enforcement        personnel. The FBI was forced to manually collect data from EPDs        from witnesses and then fly that data from Boston, Mass. to FBI        facilities in Quantico, Va.    -   5. Open Application Programming Interface (API) and Software        Development Kit (SDK) so that end customers can enhance and        extend the software themselves—as opposed to a closed,        proprietary system, or non-existent SDK or API that forces end        users to pay the developing company to extend the capability.    -   6. Secure data transmission using Triple Data Encryption        Standard (DES) or Advanced Encryption Standard (AES) 128/256        encryption for communications between the EPDs 150 and the data        server 20.    -   7. Multi-modal data transmission pathways from the EPD 150 where        data can be transmitted from the EPD 150 through either        commercial terrestrial telephony (2G, 3G, 4G, LTE, etc.), WiFi,        and/or satellite communications pathways.

Applications for the aspects of the present disclosure include:

1. Public safety

2. Emergency response

3. Crime prevention

4. Law Enforcement

5. Intelligence collection

6. Military/law enforcement hostile forces tracking

7. Military/law enforcement blue force (Agent/CI) tracking

8. Military/law enforcement mission planning

9. Military sensor planning

10. Critical installation protection

Multiple applications can be used in parallel and then combined on theserver 20.

Another exemplary embodiment of the present invention is the SNEEZESapplication, shown in FIGS. 13-16. SNEEZES is an acronym for SyndromicNotifications for Epidemics or Elevated Zone Events System. The SNEEZESapp technology combines crowd sourcing with civic responsibility tocreate an ecosystem where modern technology—specifically the billions ofdollars of investment in EPDs and the cloud—can be used to do good. Itputs technology that is already in the hands of ordinary citizens towork for the common good. Advantages of the SNEEZES app include that itcreates one central application to process and fuse multiple types ofdata from EPD and then easily send it from the EPD to the cloud with asimple buttons.

In this regard, FIG. 13 shows an exemplary SNEEZES embodiment includingEPD 1350 collecting and transmitting data as described with respect tothe previous embodiments. The SNEEZES system thus includes a datacollection, transformation and action ecosystem that includes a) an appfront end for data collection where the data is provided by end usersvoluntarily about their general health from EPD 1350, b) a hostedcloud-based enterprise service bus (ESB) for data transformation 1320,and c) a hosted web site for the publication of “heat maps” 1390 madefrom the transformed data from app users. The heat maps 1390 can beviewed and distributed to on desk top as well portable platforms(laptops, phones, and tablets). A heat map 1390 is a graphicalrepresentation of data where the individual values contained in a matrixare represented as colors. In one embodiment of SNEEZES, the heat maps1390 represent instances of people reporting feeling unwell or well.

EPD 1350 includes a sensor such as a microphone, a camera, a keyboardfor receiving text data that may be embodied by a touchscreen displayingthe keyboard, a thermometer, and a global positioning system (GPS)sensor. Further, EPD 150 may include more than one sensor, or all of thesensors listed above. In general, each EPD 150 will include at least onesensor, a processor, memory, a transmitter for transmitting the data toESB 1320, and a receiver for receiving data from ESB 1320. ESB 1320 alsoincludes a processor, a memory, a transmitter for transmitting data tothe EPD 1350, and a receiver for receiving data from EPD 1350.

People like to talk about their health. The primary SNEEZES app allowsthe public to report in near-real-time about their general health. Thatdata is then collected in the cloud and transformed into heat maps 1390.Those heat maps 1390 can then have additional data sources overlaid ontop of them to create dynamic and static views of population centers andthe general health of people around them.

Further, alerts 1310 may be sent by the ESB 1320 to the EPD 1350. Thesealerts may include health information, location information, and mayalso include advertisements to pharmacies, drug store chains, eventhosts, and/or tourist bureaus, based on how the user is feeling.

In another embodiment, an initial threshold can be set in a complexevent modeling tool within ESB 1320 such that if more than 100 uniqueSNEEZES app users report flu symptoms within 50 miles of each other itwill trigger the control system to do a web search of that area forreports of flu. If both conditions are met, ESB 1320 will send an alertto the EPD 1350 of all SNEEZES app users in the area to warn ofincreased possibility of the flu.

In still another embodiment, the SNEEZES app may have the ability toautomatically transmit body temperature data off of EPDs 1350 which canrecord body temperature. If a person in a quarantine area uses a SNEEZESenabled EPD 1350, it would allow that persons' body temperature to beautomatically recorded and forwarded through the SNEEZES app to acomplex event modeling tool within ESB 1320 and aggregated with otherSNEEZES collected data, as well as other third party collected data, togenerate alerts back to EPDs 1350 of SNEEZES App users, as well as thegeneral public.

FIG. 14 shows a display 1410 and a menu 1420 to allow a user to reporttheir health information. Screen 1430 shows that the user can remainanonymous, and may attach any of the data collected to their healthreport before sending to ESB 1320.

As shown in FIG. 15, the user can edit personal settings, which mayinclude their own competencies that they can provide the ESB 1320.

FIG. 16 shows one embodiment of a display of a heat map 1390. In FIG.16, menu bar 1610 allows a user to designate a number from 1-10 toindicate their general health. Button 1620 allows the user to enter andpost their body temperature. The heat map 1630 shows different colorsbased on health conditions in that locality. The user's location is atindicator 1640. The user can post text related to their health conditionusing button 1650, and the user can enter their heart rate using button1660.

Thus in one embodiment, a user of the EPD 1350 enters data about theirown health into EPD 1350 using the interface described above. The EPD1350 sends the data entered by the user about their health to the ESB1320. The ESB 1320 incorporates that data into heat map 1390 andtransmits the updated heat map 1390 to EPD 1350, which can then displayit for the user. Accordingly, the user can receive near-real-timeupdated heat maps providing health data covering the mapped area.Philosophy of SNEEZES=Collection−Transformation−Action

SNEEZES is an ecosystem that includes a downloadable app which connectsto a cloud based transformation engine which creates heat maps 1390 thensends machine to machine (M2M) and/or machine to person (M2P) alertswhich cause action to occur in the real world.

It was designed primarily for “everyday use” for the public tocontribute near-real-time experiential public health information to thelarger public for multiple end user purposes, including:

-   -   Get help    -   Get coupons    -   Get travel information    -   Other

Data is received by the SNEEZES back end processing capability in thecloud transforms the raw data feeds into a SNEEZES heat map 1390 whichthen shows people their information in context and allows browsers ofthe data to see the general health and wellbeing of a population areaprior to going there or for general situational awareness.

Additional RSS feed data for pollen count, heat index, health warnings,etc. would also be overlaid onto the SNEEZES heat maps 1390 to create aholistic public health snapshot informed by multiple sources includingSNEEZES users. This enhanced level of syndromic situational awarenesscould prevent issues like asthma attacks in areas where heat, pollen,and other events may trigger an attack.

The app can be customized by end users and white labeled for specificevents—Superbowl, Olympics, World Cup, Spring Break, Ski Season, MardiGras, etc.

People going on or hosting trips and/or to these venues could make useof the data as a way to show how healthy the area they are going tomight be in relation to other parts of the country.

Uses the off-the-shelf/out-of-the-box capability of the EPD to senddata—location, audio, video, text, temperature, heart rate, pulse O₂,etc. and any other data that can be collected by the EPD to the cloudfor follow on analysis, cataloguing, transformation and/or to generate aheat map.

Quick way for average citizen to share observations from their EPDsabout their general health and to see the general health of other partsof the globe.

Venue-Specific downloads are available so that at an event SNEEZES userscan let people know where they were sitting/standing etc. when an eventoccurred

SNEEZES reports can be shared with public safety and health officialseither directly to them or to a “Cut out” server they have access to.

The apps are designed to be customized so that it can be licensed to saya corporation or tourist bureau so that they become the sponsor of theapp being used at specific events such as the Olympics, etc. It can becustomized so that certain EPD features can be turned and turned off incountries where data collection of this type is prohibited.

Other advantages may include:

1. Single screen interface—as opposed to 2-4 separate applications withmultiple interfaces.

2. Multiple data feeds from multiple EPD sensors—as opposed to a userexperience where each screen can only handle one feed at a time.

3. Crowd sourced data inputs from social media—as opposed to justgetting one way alerts from an RSS feed or broadcast.

4. Can be used as a public health collection tool in real time—asopposed to collecting data and then sending it a later date in responseto an alert.

5. Open API and SDK so that end customers can enhance and extend thesoftware themselves—as opposed to a closed, proprietary, or non-existentSDK or API that forces end users to pay the developing company to extendthe capability.

Applications for SNEEZES include:

-   -   1. Public health and cohort tracking    -   2. Public safety    -   3. Emergency response    -   4. Intelligence collection    -   5. Military/law enforcement health and cohort tracking    -   6. Military sensor planning

The term cohort (as used above) effect is used in social science todescribe variations in the characteristics of an area of study (such asthe incidence of a characteristic or the age at onset) over time amongindividuals who are defined by some shared temporal experience or commonlife experience, such as year of birth, or year of exposure toradiation.

The system allows for using multiple applications in parallel and thencombining on the server/cloud side.

The present written description uses examples to disclose the presentsubject matter, including the best mode, and also to enable any personskilled in the art to practice the present subject matter, includingmaking and using any devices or systems and performing any incorporatedand/or associated methods. While the present subject matter has beendescribed in detail with respect to specific embodiments thereof, itwill be appreciated that those skilled in the art, upon attaining anunderstanding of the foregoing may readily produce alterations to,variations of, and equivalents to such embodiments. Accordingly, thescope of the present disclosure is by way of example rather than by wayof limitation, and the subject disclosure does not preclude inclusion ofsuch modifications, variations and/or additions to the present subjectmatter as would be readily apparent to one of ordinary skill in the art.

The invention claimed is:
 1. A method of controlling a device includinga processor operatively coupled to a display, a wireless transceiver andat least one sensor to surreptitiously capture and wirelessly streamaudio and/or visual information, the method comprising: (a) receiving atleast one command input by a user of the device; (b) in response toreceipt of the at least one command, controlling the device to display ascreen that masquerades device operation by concealing the capturing andwirelessly streaming and instead shows a user interactive function thatis unrelated to the capturing and wirelessly streaming, and (c)wirelessly streaming the captured audio and/or visual information to aserver while the screen is displayed, wherein the user interactivefunction is a function of a user interactive application, the displayedscreen shows an image comprising data, the displayed data having anappearance that changes automatically and dynamically in real-time, andthe at least one sensor comprises a video camera and/or a microphone. 2.The method according to claim 1, wherein the device is a portablehandheld wireless device including a handheld housing.
 3. The methodaccording to claim 1, wherein the device is a desktop or laptopcomputer.
 4. The method according to claim 1, wherein the at least onesensor comprises a video camera and/or a microphone.
 5. The methodaccording to claim 1, wherein the user interactive function is agambling function of a slot machine game application.
 6. The methodaccording to claim 1, wherein the displayed screen shows three or morereels and at least one button.
 7. The method according to claim 1,wherein the displayed screen shows “7” and “BAR” symbols.
 8. The methodaccording to claim 1, wherein the captured audio and/or visualinformation is streamed to a server for follow-on analysis anddistribution by the server.
 9. The method according to claim 1, whereinthe user interactive function is a gambling function of a slot machinegame application, the displayed screen shows an image comprising threeor more reels, at least one button, and “7” and “BAR” symbols, and theat least one sensor comprises a video camera and/or a microphone.
 10. Anon-transitory computer-readable storage medium having stored therein aninformation processing program for execution by a device comprising aprocessor operatively coupled to a display, a wireless transceiver andat least one sensor, the information processing program comprisinginstructions that, when executed by the processor, causing the device toperform operations including: receiving at least one command input by auser of the device; in response to receipt of the at least one command,generating, for display on the display, a screen that masquerades deviceoperation by concealing capturing and wirelessly streaming audio and/orvisual information and instead shows a user interactive function that isunrelated to the capturing and wireles sly streaming, and wireles slystreaming the captured audio and/or visual information to a server whilethe screen is displayed, wherein the user interactive function is afunction of a user-interactive application, the displayed screen showsan image comprising data, the displayed data having an appearance thatchanges automatically and dynamically in real-time, and the at least onesensor comprises a video camera and/or a microphone.
 11. A devicecomprising a processor operatively coupled to a display, a wirelesstransceiver and at least one sensor, the processor being configured to:receive at least one command input by a user of the device; in responseto receipt of the at least one command, display a screen thatmasquerades device operation by concealing capturing and wirelesslystreaming audio and/or visual information and instead shows a userinteractive function that is unrelated to the capturing and wirelesslystreaming, and wirelessly stream the captured audio and/or visualinformation to a server while the screen is displayed, wherein the userinteractive function is a function of a user-interactive application,the displayed screen shows an image comprising data, the displayed datahaving an appearance that changes automatically and dynamically inreal-time, and the at least one sensor comprises a video camera and/or amicrophone.
 12. The device according to claim 11, wherein the device isa portable handheld wireless device including a handheld housing. 13.The device according to claim 11, wherein the user interactive functionis a gambling function of a slot machine game application, the displayedscreen shows an image comprising three or more reels, at least onebutton, and “7” and “BAR” symbols, and the at least one sensor comprisesa video camera and/or a microphone.
 14. A system for collecting datafrom at least one device, the system comprising: the device according toclaim 11; and a server including a receiver configured to receive thedata collected by the at least one sensor of the device, and a processorconfigured to issue one or more commands to the device to requestcollecting audio and/or visual information by the at least one sensor ofthe device.
 15. The method according to claim 1, wherein the device is aremote device, and the captured audio and/or visual information can beencrypted on the remote device to maintain privacy and security of boththe captured audio and/or visual information and the remote device atrest or in motion.
 16. The method according to claim 1, wherein the dataincludes a number and/or an icon whose appearance changes automaticallyand dynamically in real time.
 17. The method according to claim 1,wherein the user interactive application is a gaming application.